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Abstract 



Collective intelligence, which aggregates the 
shared information from large crowds, is of- 
ten negatively impacted by unreliable infor- 
mation sources with the low quality data. 
This becomes a barrier to the effective use 
of collective intelligence in a variety of ap- 
plications. In order to address this issue, 
we propose a probabilistic model to jointly 
assess the reliability of sources and find the 
true data. We observe that different sources 
are often not independent of each other. In- 
stead, sources are prone to be mutually in- 
fluenced, which makes them dependent when 
sharing information with each other. High 
dependency between sources makes collective 
intelligence vulnerable to the overuse of re- 
dundant (and possibly incorrect) information 
from the dependent sources. Thus, we re- 
veal the latent group structure among depen- 
dent sources, and aggregate the information 
at the group level rather than from individual 
sources directly. This can prevent the collec- 
tive intelligence from being inappropriately 
dominated by dependent sources. We will 
also explicitly reveal the reliability of groups, 
and minimize the negative impacts of unre- 
liable groups. Experimental results on real- 
world data sets show the effectiveness of the 
proposed approach with respect to existing 
algorithms. 



1. Introduction 

Collective intelligence aggregates contributions from 
multiple sources in order to collect data for a variety 
of tasks. For example, voluntary participants collabo- 
rate with each other to create a fairly extensive set of 
entries in Wikipedia; a crowd of paid persons may per- 
form image and news article annotations in Amazon 
Mechanical Turk. These crowdsourced tasks usually 
involve multiple objects, such as Wikipedia entries and 
images to be annotated. The participating sources col- 
laborate to claim their own observations, such as facts 
and labels, on these objects. Our goal is to aggregate 
these collective observations to infer the true values 
(e.g., the true fact and image label) for the different 
objects (Zhao et al., 2012; Pasternack & Roth, 2010; 
Galland et al., 2010). 

We note that an important property of collective in- 
telligence is that different sources are typically not in- 
dependent of one another. For example, in the same 
social community, people often influence each other, 
where their judgments and opinions are not indepen- 
dent. In addition, task participants may obtain their 
data and knowledge from the same external informa- 
tion source, and their contributed information will be 
dependent. Thus, it may not be advisable to treat 
sources independently and directly aggregate the in- 
formation from individual sources, when the aggrega- 
tion process is clearly impacted by such dependencies. 
In this paper, we will infer the source dependency 
by revealing latent group structures among involved 
sources. Dependent sources will be grouped, and their 
reliability is analyzed at the group level. The incorpo- 
ration of such dependency analysis in group structures 
can reduce the risk of overusing the observations made 
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by the dependent sources in the same group, especiahy 
when these observations are unrehable. This helps pre- 
vent dependent sources from inappropriately dominat- 
ing collective intelligence especially when these source 
are not reliable. 

Moreover, we note that groups are not equally reliable, 
and they may provide incorrect observations which 
conflict with each other, either unintentionally or ma- 
liciously. Thus, it is important to reveal the reliability 
of each group, and minimize the negative impact of 
the unreliable groups. For this purpose, we study the 
general reliability of each group, as well as its spe- 
cific reliability on each individual object. These two 
types of reliability are closely related. General reli- 
ability measures the overall performance of a group 
by aggregating each individual reliability over the en- 
tire set of objects. On the other hand, although each 
object-specific reliability is distinct, it can be better 
estimated with a prior that a generally reliable group 
is likely to be reliable on an individual object and 
vice versa. Such prior can reduce the overfitting risk 
of estimating each object-specific reliability, especially 
considering that we need to determine the true value 
of each object at the same time (Kasneci et al., 2011; 
Bachrach et al., 2012). 

The remainder of this paper is organized as follows. In 
Section 2, we formally define our problem and nota- 
tions in the paper. The Multi-Source Sensing (MSS) 
model for the problem is developed in Section 3, fol- 
lowed by the group observation models in Section 4. 
Section 5 presents the inference algorithm. Then we 
evaluate the approach in Section 6 on real data sets, 
and conclude the paper in Section 7. 

2. Problem and Notational Definitions 

We formally define the following Mult i- Source Sens- 
ing (MSS) model which abstracts the description of 
collective intelligence. Suppose that we have a set 
S := {5'i,5'2,--' ^Sn} of sources, and a set O := 
{Oi,02,''' ^Om} of M objects. Each object Om 
takes a value tm from a domain AVn which describes 
one of its attributes. Each source Sn in S reports 
its observation yn,m ^ on an object Om- Then 
the goal of the MSS model is to infer the true value 
tm of each object Om from the observations made by 
sources. 

In this paper, we are particularly interested in categor- 
ical domain Xm = {1, • " ' ^^m} with discrete values. 
For example, in many crowdsourcing applications, we 
focus on the (binary-valued) assertion correctness in 
hypothesis test and (multi-valued) categories in clas- 



sification problem. However, the MSS model can be 
straightly extended to continuous domain. Due to the 
space limitation, we leave this topic in the extended 
version of this paper. 

Figure 1 illustrates an example, where five sources 
make their observations on four objects. An object 
can be an image or a biological molecule, and an anno- 
tator or a biochemical expert (as a source) may claim 
the category (as the value) for each object. Alterna- 
tively, an object can be a book, and a book seller web 
site (as a source) claims the identity of its authors (as 
the values). In a broader sense, objects are even not 
concrete objects. They can refer to any crowdsourced 
tasks, such as questions (e.g., "is Peter a musician?") 
and assertions (e.g., "George Washington was born on 
February 22, 1732." and "an animal is present in an 
image,"), and the observations by sources are the an- 
swers to the questions, or binary- valued positive or 
negative claims on these assertions. 

It is worth noting that each source does not need to 
claim the observations on all objects in O. In many 
tasks, sources make claims only on small subsets of 
objects of interest. Thus, for notational convenience, 
we denote all claimed observations by y in bold, and 
use / = {(n, m)|3 yn,m ^ y} to denote all the indices in 
y. We use the notations i^,- = {Tn\3 (n^m) G /} and 
I -,771 = (n, m) G /} to denote the subset of indices 
that are consistent with the corresponding subscripts 
n and m. 

Meanwhile, to model the dependency among sources, 
we assume that there are a set of latent groups 
{Gi, 6^2, • • • I? aiid each source Sn is assigned to one 
group Gg^ where gn G {1,2,---} is a random vari- 
able indicating its membership. For example, as il- 
lustrated in Figure 1, the five sources are inherently 
drawn from two latent groups, where each source is 
linked to the corresponding group by dotted lines. 
Each latent group contains a set of sources which are 
infiuenced by each other and tend to make similar ob- 
servations on objects. The unseen variables of group 
membership will be inferred mathematically from the 
underlying observations. Here, we do not assume any 
prior knowledge on the number of groups. The com- 
position of these latent groups will be determined with 
the use of a Bayesian nonparametric approach by stick- 
breaking construction (Sethuraman, 1994), as to be 
presented in the next section. 

To minimize the negative impact of unreliable groups, 
we will explicitly model the group-level reliability. 
Specifically, for each group Gi, we define a group re- 
liability score ui G [0, 1] in unit interval. This value 
measures the general reliability of the group over the 
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entire set of objects. The higher value of ui indicates 
the greater reliability of the group. Meanwhile, we also 
specify the rehability ri^rn ^ {0? 1} of each group Gi on 
each particular object Om- When ri^rn = 1, group Gi 
will have reliable performance on O^, and otherwise 
it will be unreliable. In the next section, we will clar- 
ify the relationship between general reliability ui and 
object-specific reliability ri^rn- 

3. Multi-Source Sensing Model 

In this section, we present a generative process for 
the multi-source sensing problem. It defines a group 
reliability structure to find the dependency between 
sources at the same time when we infer their reliabil- 
ity at the group level. 

First we define the following generative model for 
multi-source sensing (MSS) process below, the details 
of which will be explained shortly. 

A-GEM(a>:), ^/n|A - Discrete(A), (1) 

ui ^ Beta(6i,6o),n,^ ^ BeTn{ui),tm ^ Unif (2) 

7Tl^m\ri,m,tm = Z ^ Hri^^{tm) (3) 
yn,m\7ri,m,gn ^ FilZg^^rn) (4) 

forn = 1, 2, • • • , TV, m = 1, 2, • • • , M, / = 1, 2, • • • . Fig- 
ure 2 illustrates the generative process in a graph- 
ical representation. Here, ^n|A ~ Discrete(A) de- 
notes a discrete distribution, which generates the value 
Qn = i with probability A^; Beta, Bern and Unif stand 
for Beta, Bernoulli and uniform distributions, respec- 
tively. We explain the detail of this generative process 
below. 



In Eq. (1), we adopt the stick-breaking construc- 
tion GEM(/^) (named after Griffiths, Engen and Mc- 
Closkey) with concentration parameter n G to de- 
fine the prior distribution of assigning each source Sn 
to a latent group Gg^ (Sethuraman, 1994). Specif- 
ically, in GEM(/^), a set of random variables p = 
{pi,P2,-''} are independently drawn from the Beta 
distribution pi ~ Beta(l,Ai:). They define the mixing 
weights A of the group membership component such 
that p{gn = l\p) = Xi = pi n!ll (1 - Pi)' Obviously, 
by the above stick-breaking process, we do not need 
the prior knowledge of the number of groups. This 
number will be determined by capturing the degree of 
dependency between sources. 

Clearly, we can see that the parameter tz in the above 
GEM construction plays the vital role of determining 
a priori the degree of dependency between sources. 
Actually, according to the GEM construction, we can 
verify that the probability of two sources Sn and Sm 
being assigned to the same group is 

+00 

P{9n = gm) = ^EP(^n = /|A)P(^^ = /|A) 
1 = 1 ^ 

+ 00 +00 l — l 

We can find that when ti is smaller, source are more 
likely to be assigned to the same group where they 
are dependent and share the same observation model. 
This will yield higher degree of dependency between 
sources. As n increases, the probability that any two 
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sources belong to the same group will decrease. In the 
extreme case, as +00, this probability will ap- 

proach to zero. In this case, all sources will be assigned 
to distinctive groups, yielding complete independence 
between sources. This shows that the model can flexi- 
bly capture the various degree of dependency between 
sources by setting an appropriate value of hz. 

In Eq. (2), we define a Beta distribution Beta(6i,6o) 
on the group reliability score ui^ where bi and bo are 
the soft counts which specify whether a group is reli- 
able or not a priori, respectively. Then object-specific 
reliability n,rn ^ {^A} is sampled from the Bernoulli 
distribution BeTn{ui) to specify the group reliability on 
a particular object Om- We can find that the higher 
the general reliability ui^ the more likely Gi is reliable 
on a particular object Om with ri^rn being sampled 
to be 1. This suggests that a generally more reliable 
group is more likely to be reliable on a particular ob- 
ject. In this sense, the general reliability serves as a 
prior to reduce the overfitting risk of estimating object- 
specific reliability in MSS model. 

In Eq. (2), we adopt uniform distribution as the prior 
on the true value tm of each object over its domain 
Xm- The uniform distribution sets an unbiased prior 
so that true values will be completely determined a 
posteriori given observations in the model inference. 

Eq. (3) and Eq. (4) define the generative process 
for the observations of each source in its assigned 
group. Specifically, given the group membership 
each source Sn generates its observation yn,m accord- 
ing to the corresponding group observation model 
F{7Vg^^m). The TVi^rn of this modcl is drawn from the 
conjugate prior Hn ^(^m) which depends on the true 
value tm and the object-specific group reliability ri^m- 
In the next section, we will detail the specification of 
Hri^^{tm) and F{7Vi^m) in categorical domain. The 
models in other domain can be obtained by adopt- 
ing the corresponding distribution with the analogous 
idea. 

4. Group Observation Models 

In categorical domain, for each group, we choose the 
multinomial distribution Fijzi^m) = Mult(7r/^rn) as its 
observation model to generate observations yn,m for 
its member sources. Its parameter tt/ m is generated 
by: 



'^l,'m\'^l,m') — Z Hfj^^^itm^ 

:= Dir(6'(''''"\---,r?('''-'"\--- ,6'(''''")) 
^ « ' 4- 

Z—1 zth entry 



where Dir denotes Dirchlet distribution, and 
and 77^^^"^) are its soft counts for sampling the false 
and true values under different settings of vi^m- Below 
we will explain how to set these soft counts under these 
settings. 

For a reliable group Gi on object Om (i-e., ri^m = 1), it 
should be more likely to sample the true value tm = z 
as its observation than sampling any other false values. 
Thus, we should set a larger value for 7^^^^"^) than for 

On the other hand, if group Gi is unreliable on object 
Om (i-e., ri^m = 0), we can distinguish between care- 
less and malicious groups, and set their parameters in 
different ways: 

I. careless group: We define Gi as a careless group, 
whose member sources randomly claim values for ob- 
ject Om, no matter which value is true. In this case, an 
equal soft count is set for the true and false values, i.e., 
Qin^rn) _ ^in^rn) ^ Thls wlll makc the true value indis- 
tinguishable from the false ones, so that the member 
sources makes a random guess of the true value. 

II. malicious group: In this case, group Gi contains 
malicious sources which intentionally provide mislead- 
ing information about the true value of object Om- In 
other words, the group tends to claim the false val- 
ues for object Om, and thus we should set a larger 
value for ^(^^"^) than for 7^(^^"^). Such malicious group 
can still contribute certain information if we read its 
observations in a reverse manner. Actually, by set- 
ting 0^'^^^^^ > Tf^^hrn) ^ MSS model gives the un- 
claimed observations larger weight (corresponding to 
larger value of 6>(^^"^)) to be evaluated as the true 
value. 

5. Model Inference 

In this section, we present the inference and learning 
processes. The MSS model defines a joint distribu- 
tion on g = {gn}, r {n,m}, u {ui}, t {tm}, 
TT = {T^i^m} and the source observations y. We wish 
to infer the tractable posterior p(g, r, u, t, 7r|y) with 
a parametric family of variational distributions in the 
factorized form: 

^(g,r,u,t,7r) = ]\q{gn\^n) W Q{n ,171 \'T~l,m^ 

n l,m 
I m l^m 

with parameters cp^, Ti^m, /^z, and cxi^m for these 
factors. The distribution and the parameter for each 
factor can be determined by variational approach 
(Jordan et al., 1999). Specifically, we aim to maxi- 
mize the lower bound of the log likelihood logp(y). 
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i.e., C{q) = Eglnp(g,r,u,t,7r,y) - EI((7(g,r,u,t,7r)) 
with the entropy function EI(-) to obtain the optimal 
factorized distribution. The lower bound can be max- 
imized over one factor while the others are fixed. This 
is an approach which is similar to coordinate descent. 
All the factors are updated sequentially over steps un- 
til convergence. We derive the details of the steps for 
updating each factor below. 

1: Update each factor q{7Zi^rn\oii^m) for the group ob- 
servation parameter izi^rn- By variational approach, 
we can verify that the optimal q{'Ki^m\oLi^m) has the 
form 

q{'ni^m\oLi^m) cxexp{ E lnp(7r^,,n|n,m, ^m) 

+ E \np{yn^m\^l^m,gn)} 



nei., 



It still has Dirichlet distribution with the parameters 

nel.,m 

ri,rr^e{0,l} 

for each k G A'^, where S {A} is the indicator function 
which outputs 1 if holds, and otherwise. Here 
we index the element in cxi and tt/ by k after the 
colon. We will follow this notation convention to index 
the element in vectors in this paper. 

2: Update each factor q{ui\(5i) for general group reli- 
ability ui. We have 

\iiq{ui\(5i) oc E \np{ri^rn\ui) + lnp(^i/|6i, 6o) 



m 

qoin^m) + ^0 - 1) ln(l - ui) 



where qi{ri^rn) is short for q{ri^rn = i) for ^ = 0,1, 
respectively. We can find the posterior of ui still has 
Beta distribution as Beta(/3^) with parameter 



We can find that the above updated parameter sums 
up the posterior reliability qiiri^rn) and qo{ri^rn) over 
all objects. This corresponds with the intuition that 



the general reliability is the sum of the reliability on 
individual objects. 

3: Update each factor q{ri^rn\Ti^rn) for the object- 
specific reliability ri^rn of group Gi on Om' 



,171 |'^^,m7 ^m) 

qitm),qi^i,m) 



- E \nq{ri^rn\ui) 



(6) 



Thus, we have 

^^q{ri,m\Tl,m) 

(X 



V q{tm = - 1) E InTTi^m^k 



kex, 



(7) 



-^n^rn E \nUi ^ {I - n^rn) E - Ui) 

q{ui) q{ui) 

for ri^rn ^ {0, 1}, respectively. Here we compute the 
expectation of the logarithmic Dirichlet variable as 

with the digamma function the expectation of 

the logarithmic Beta variables 



Eg(^^)lnii| = + P1.2) - 



and 



E,(,^)ln(l - ui) = + pi,2) - 

Finally, the updated values of q{ri^rn) are normalized 
to be valid probabilities. 

The last line of Eq. (7) reflects how the general reli- 
ability ui affects the estimation of the object-specific 
reliability. This embodies the idea that a generally reli- 
able group is likely to be reliable on a particular object 
and vice versa. This can reduce the overfitting risk of 
estimating ri^ra especially considering that qitm) in the 
second line also need to be estimated simultaneously 
in MSS model as in the next step. 

4: Update each factor qitmWm) for the true value. 
We have 

\nqitm = k\vm) cx lnp{tm k) 

+ ^ q{ri,m) E \np{7ri 

I n,^G{0,i} 

This suggests that 
\nq{tm = k\um) 



q{.T^l,m,) 



k'^k 



1) E \rilli^rn-k'} 

q{T^l,m) 
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All q{tm = k)^k G ^rn are normalized to ensure they 
are validate probabilities. 

5: Update each factor q{gn\^n) ^^r the group assign- 
ment of each source. We can derive 

\nq{gn l\(pj 

OC E lnp(^n=/|p)+ ^ E \np{yn,m\7Tl,m,gn = I) 

— E lnp(^n = /|p) + ^ E ln7r;,m;y^^ 

This shows that q{gn = l\^n) ^ multinomial distri- 
bution with its parameter as 

/ /I A exp{Un,i) 

^n-,l = q[gn = l\^n) = ^ (8) 

exp(/7n,0 

where 

Un,i= E \np{gn =l\p) ^ E Ihtt^,^;^^^^ 



As in (Kurihara et al., 2006), we truncate after L 
groups: the posterior distribution q{pi) after the level 
L is set to be its prior p{pi) from Beta(l,A>:); and all 
the expectations E Inni^rn^k after L are set to: 

^g(7rz,^) ln7r^,^;/e = E { E[\n7ri^rn;k\ri,m,tm] } 

with p{ri^rn) defined as in Eq. (2) for all / > L, re- 
spectively. The inner conditional expectation in the 
above is taken with respect to the probability of TVi^rn 
conditional on ri^rn and tm as defined in (3). Simi- 
lar to the family of nested Dirichlet process mixture 
in (Kurihara et al., 2006), this will form a family of 
nested priors indexed by L for the MSS model. Thus, 
we can compute the infinite sum in the denominator 
of Eq. (8) as: 

E"^ nr \ exp([/n,L+i) 
. ^ ^^P^^-'^ = l-exp( E 



l=L-\-l 



Pi~Beta(l,/^) 



6: Finally, we can find that before the truncation level 
L, the posterior distribution q{pi) ~ Beta((/)i^i, (/)^^2) is 
updated as 

N N oo 

n—1 n—lj—i-\-l 

The above steps are iterated to yield the optimal fac- 
tors. 



6. Experimental Results 

In this section, we compare our approach with other 
existing algorithms and demonstrate its effectiveness 
for inferring source reliability together with the true 
values of objects. The comparison is performed on a 
book author data set from online book stores, and a 
user tagging data set from the online image sharing 
web site Flickr . com. 

Book author data set: The first data set is the 
book author data set prepared in (Yin et al., 2007). 
The data set is obtained by crawling 1,263 com- 
puter science books on AbeBooks.com. For each book, 
AbeBooks.com returns the book information extracted 
from a set of online book stores. This data set con- 
tains a total of 877 book stores (sources), and 24,364 
listings of books (objects) and their author lists (ob- 
ject values) reported by these book stores. Note that 
each book has a different categorical domain A' that 
contains all the authors claimed by sources. 

Author names are normalized by preserving the first 
and last names, and ignoring the middle name of each 
author. For evaluation purposes, the authors of 100 
books are manually collected from the scanned book 
covers (Yin et al., 2007). We compare the returned 
results of each model with the ground truth author 
fists on this test set and report the accuracy. 

We compare the proposed algorithm with the fol- 
lowing ones: (1) the naive Voting algorithm which 
counts the top voted author list for each book as 
the truth; (2) TruthFinder (Yin et al., 2007); (3) 
Accu (Dong et al., 2009) which considers the depen- 
dency between sources; (4) 2-Estimates as described 
in (Galland et al., 2010) with the highest accuracy 
among all the models in (Galland et al., 2010) (5) 
MSS, which is our proposed algorithm. In the exper- 
iments, we choose the parameters 77^^^"^) and 6>(^^"^) 
from {1.0, 2.0, 5.0, 10.0} for n,^ e {0, 1} as in Section 
4, bo, bi from {1.0, 2.0, 4.0}, and from {1.0, 5.0, 10.0}. 
Due to the unsupervised nature of the problem, we 
pick the set of parameters with the maximum obser- 
vation likelihood. 

Table 2 compares the results of the different algorithms 
on the book author data set in terms of the accuracy. 
The MSS model achieves the best accuracy among all 
the compared models. We note that the proposed MSS 
model is an unsupervised algorithm which does not in- 
volve any training data. That is to say, we do not use 
any true values in the MSS algorithm in order to pro- 
duce the reliability ranking as well as other true values. 
Even compared with the accuracy of 0.91 of the Semi- 
Supervised Truth Finder (SSTF) (Yin & Tan, 2011) 
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Table 1. Top- 10 and bottom- 10 book stores ranked by their posterior probability of belonging to a reliable group. We 
also report the accuracy of these bookstores on the test set. 



top- 10 bookstore 


accuracy 


bottom- 10 bookstore 


accuracy 


International Books 


1 


textbooksNow 


0.0476 


happybook 


1 


Gunter Koppon 


0.225 


eCampus.com 


0.9375 


www.textbooksrus.com 


0.3333 


COBU GmbH & Co. KG 


0.875 


Gunars Store 


0.2308 


HTBOOK 


1 


Indoo.com 


0.3846 


AlphaCraze.com 


0.8462 


Bobs Books 


0.4615 


Cobain LLC 


1 


OPOE-ABE Books 





Book Lovers USA 


0.8667 


The Book Depository 


0.3043 


Versandantiquariat Robert A. Mueller 


0.8158 


Limelight Bookshop 


0.3896 


THESAINTBOOKSTORE 


0.8214 


textbookxdotcom 


0.4444 



Table 2. Comparison of different algorithms on book author and Flickr data set. On book author data set, the algorithms 
are compared by their accuracies. On Flickr data set, the algorithms are compared by their average precisions and recalls 
on 12 tags. 



Model 


book author data set 


FHckr data set 


accuracy 


precision 


recall 


Voting{Doiig et al., 2009) 


0.71 


0.8499 


0.8511 


2-Estimates{Qdl\d.iid et al., 2010) 


0.73 


0.8545 


0.8602 


TruthFinder{Ym k Tan, 2011) 


0.83 


0.8637 


0.8649 


Accu{Bong et al., 2009) 


0.87 


0.8731 


0.8743 


MSS 


0.95 


0.9176 


0.9212 



0.95 

0.9 

I 0.85 

i 0.8 
< 




0.5 1 2 5 8 10 12 15 20 



K 

Figure 3. Model accuracy versus different k on book author 
dataset. 



using extra training data with known true values on 
some objects, the MSS model still achieves the highest 
0.95 accuracy. It suggests that with additional train- 
ing data, the MSS model may improve its accuracy 
further. 

Since is predicative of the dependency between 
sources, we study the changes of the model accuracy 
versus various in Figure 3. We know that when 
= 0, all sources are completely dependent, and as- 
signed to the same group. At this time, the model has 
a much lower accuracy, since all sources are tied to the 
same level of reliability within a single group. As hi 



increases, the accuracy achieves the peak at k = 5.0. 
After that, it deteriorates as the model gradually stops 
capturing the source dependency with increased tz. 
This demonstrates the importance of modeling the 
source dependency, and the capability of MSS model 
to capture such dependency by k,. 

Moreover, to compare the reliability between sources, 
we can define the reliability of each source Sn by the 
expected reliability score of its assigned groups as 

Reliability(5'n) = ^q{gn = E [ui] 

I Qiui\f3i) 

where 



Then, sources can be ranked based on such source reli- 
ability. In Table 1, we rank the top- 10 and bottom- 10 
book stores in this way. In order to show the extent to 
which this ranking list is consistent with the real source 
reliability, we provide the accuracy of these bookstores 
on test data sets. Note that each individual bookstore 
may only claim on a subset of books in the test set, and 
the accuracy is computed based on the claimed books. 
From the table, we can see that the obtained rank of 
data sources is consistent with the rank of their accu- 
racies on the test set. On the contrary, the accuracy 
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Table 3. The rounds used before convergence and computing time for each model. 



Model 


Bookstore 


User Tagging 


Rounds 


Time(s) 


Rounds 


Time (s) 


Voting 


1 


0.2 


1 


0.5 


2-Estimates 


29 


21.2 


32 


628.1 


TruthFinder 


8 


11.6 


11 


435.0 


Accu 


22 


185.8 


23 


3339.7 


MSS 


9 


10.3 


12 


366.2 









(a) balloon 


(b) snow leopard 









(c) guitar (d) pumpkin 



Figure 4. Examples of image and the associated user tags 
in Flickr data set. In each subfigure the left image is 
correctly tagged by users, while the right one is wrongly 
tagged. 



of the bottom- 10 bookstores is much worse compared 
to that of the top-10 book stores on the test set. This 
also explains partly the better performance of the MSS 
model. 

User tagging data set: We also evaluate the algo- 
rithm on a user tagging data set from an online im- 
age sharing web site Flickr.com. This data set con- 
tains 13, 528 users (data sources) who annotate 36, 280 
images (data objects) with their own tags. We con- 
sider 12 tags - "balloon," "bird," "box," "car," "cat," 
"child," "dog," "flower," "snow leopard," "waterfall," 
"guitar," "pumpkin" for evaluation purposes. Each 
tag is associated with a binary value 1/0 to represent 
its presence or not in an image, and we apply MSS 
model to these 12 tags separately to find whether they 
are present on each image. To test accuracy, we man- 
ually annotate these 12 tags on a subset of 1, 816 im- 
ages. Figure 4 illustrates some image examples in this 
data set and the tags annotated by users. We can find 
some images are wrongly tagged by users. The MSS 
model aims to correct these errors and yield accurate 
annotations on these images. 

We follow the same experimental setup as on the book 
author data set. Table 2 shows the average precision 
and recall on the 12 tags by the compared algorithms. 
We can see that MSS still performs the best among 
these compared algorithms. 



We also compare the computational time used by dif- 
ferent algorithms in Table 3. The experiments are con- 
ducted on a personal computer with Intel Core 17-2600 
3.40 GHz CPU, 8 GB physical memory and Windows 
7 operating system. We can see that compared with 
most of other algorithms, MSS model can converge in 
fewer rounds with less computational cost. 

7. Conclusion 

In this paper, we propose an integrated true value 
inference and group reliability approach. Dependent 
sources which are grouped together, and their (gen- 
eral and specific) reliability is assessed at group level. 
The true data values are extracted from the reliable 
groups so that the risk of overusing the observations 
from dependent sources can be minimized. The overall 
approach is described by a probabilistic multi-source 
sensing model, based on which we jointly infer group 
reliability as well as the true values for objects a pos- 
terior given the observations from sources. The key to 
the success of this model is to capture the dependency 
between sources, and aggregate the collective knowl- 
edge at the group granularity. We present experimen- 
tal results on two real data sets, which demonstrate 
the effectiveness of the proposed model over other ex- 
isting algorithms. 
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